perf(fair2): vectorise forcing-aggregation output extraction (~9x at 842 members)#109
Merged
benmsanderson merged 2 commits intoJul 2, 2026
Conversation
Profiling a native-calibration ensemble run (py-spy, ssp245, 400-842 members) showed ~68% of wall time in the output extractor's forcing aggregations -- specifically repeated xarray `.sel(specie=...).isel(...)` orthogonal indexing, run once per (scenario, member) for every aggregation. FaIR's actual integration was ~6%, and the conc-driven `unstep_concentration` path ~0.3% (so conc-vs-emissions is not the cost). Materialise `f.forcing` to a numpy array once (species->index map) and do the per-member sums / single-species lookups with integer indexing. Same species order and summation order, so results are numerically identical. Benchmarks (ssp245, native AR7 1.6.0 calibration, single core): members before after 200 23.9 s 14.4 s 400 63.9 s 21.5 s 842 ~370 s 39.6 s (~9x; superlinear scaling removed) GSAT, ERF|Greenhouse Gases and ERF|Aerosols (ensemble mean AND std) match the previous xarray path to 6 d.p. Adds a unit equivalence test (numpy aggregation == direct xarray reduction) and the existing modern-adapter snapshot is unchanged. No FaIR internals touched. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
benmsanderson
merged commit bdeb4e7
into
feat/fair2-ciceroscmpy2-adapters-and-runmode-nonfork
15 checks passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
FaIR2 native-calibration ensemble runs were the throughput bottleneck for the AR7 Ch5 chapter (~370 s/pathway at 842 members, single core, superlinear). Profiling (py-spy, ssp245, 400–842 members) pinned the cost:
.sel(specie=…).isel(scenario, config)orthogonal indexing, run once per (scenario, member) for every aggregation recipe.unstep_concentration: ~0.3%.So the hotspot is Python/xarray output handling, not FaIR's solver and not the conc-driven path (measured conc-driven 14.3 s ≈ emissions-driven 14.5 s at 100 members).
Fix
Materialise
f.forcingto a numpy array once with a species→index map, and do the per-member sums / single-species lookups with integer indexing instead of per-member xarray orthogonal indexing. Same species order and summation order → numerically identical.Benchmarks (ssp245, native AR7 1.6.0 calibration, single core)
Superlinear scaling is removed (now ~linear in members).
Equivalence
GSAT, ERF|Greenhouse Gases and ERF|Aerosols (ensemble mean and std) match the previous xarray path to 6 d.p. Added a unit equivalence test (
tests/unit/adapters/test_fair2_output_extractor.py) asserting the numpy aggregation path equals a direct xarray reduction; the existingtest_modern_adapters.pyFaIR2 snapshot is unchanged. No FaIR internals touched, MAGICC/CICERO/conc-driven paths untouched.🤖 Generated with Claude Code